test: automate WPT updates with drivers in test/wpt #24035

joyeecheung · 2018-11-02T10:08:02Z

This is a big diff but note that the biggest commit (the first one) is automated by git node wpt. For a high-level design see test/wpt/README.md in this diff.

There is still work to do with the upstream to port more tests to .js files and split them properly so that we only need to skip the parts that truely cannot be run by us (e.g. ones that requires Response/Request objects).

Currently there are still a few test-whatwg-* tests in test/parallel that are copy-pasted with manually added fixtures/linter switches, but I think we can land these patches first to eliminate the manual maintenance of of the tests that can be migrated, and gradually migrate the rest of them later while working with the upstream.

Refs: #23192

Current status:

console tests: all migrated here
URL tests:
- Some of them are still in .html files upstream
- Some of them mixes JS tests and <a>/<area> tests with references to document.createElement in .window.js, or needs something like Request/Response which we don't have
- There are a bunch of test-whatwg-url-custom-* tests that can be upstreamed and pulled here later
- Others are migrated here
encoding tests: I have a WIP branch for them - but will need to figure out the ICU requirements in the CI first as some of the subtests need more support than the others, and we will probably want to split them into separate files somehow in the upstream as well, so I'd prefer to leave them in a subsequent PR

Note that none of the IDL tests currently pass - I plan to fix them later after this PR lands.

test: use git node wpt to pull WPT into test/fixtures

This patch uses the git node wpt command in node-core-utils
to initialize test/fixtures/wpt with a subset of the Web
Platform Tests (https://github.com/web-platform-tests/wpt),
including:

console tests
url tests
interfaces (.idl WebIDL files)
test harness located in resources, including the WebIDL
parser

Also initializes the README.md, LICENSE and versions.json in the
test/fixtures/wpt.

In later patches, we will use these files to run WPT
and move away from the manually maintained copy-pasted files
in test/parallel/test-whatwg-*.

https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt

test: remove WPT tests that are now .any.js in the upstream

In favor of actual .any.js tests that will be run with the WPT
harness in a subsequent patch.

The other types of .js tests (e.g. .window.js tests) and the .html
tests remained to be supported or migrated in the upstream.

test: use URL fixtures under test/fixtures/wpt/url/resources

Removes the following files:

test/fixtures/url-tests.js
test/fixtures/url-setter-tests.js
test/fixtures/url-toascii.js

in favor of:

test/fixtures/wpt/url/resources/urltestdata.json
test/fixtures/wpt/url/resources/setters_tests.json
test/fixtures/wpt/url/resources/toascii.json

Also removes dependency of fixtures/url-tests.js in http2 tests
and use fixtures/person-large.jpg instead since they are just
looking for a big enough file to transfer.

test: initialize test/wpt to run URL and console .js tests

This patch:

Creates a new test suite wpt that can be used to run a subset
of Web Platform Tests
Adds a WPTRunner in test/common/wpt.js that can run the WPT
subset in test/fixtures/wpt with a vm and the WPT harness
while taking the status file in test/wpt/status into account.
Here we use a new format of status file (in JSON) to handle specific
requirements (like ICU requirements) in the tests and to handle
expected failures and TODOs.
Adds documentation on how the runner and the update automation works
Runs the WHATWG URL tests and the console tests with the new test
runner.

With this patch we eliminates the need of copy-pasting with manual
modifications to update a large chunk of our WPT subset previously
maintained in test/parallel. Now the tests run in test/wpt can
be automatically updated with git node wpt without modifications
by the actual WPT harness instead of our home-grown mock.

There are still a few URL tests left that need to be migrated in the
upstream to be placed in .js instead of .html - we currently still use
the legacy harness mock in the test files.

doc: remove legacy WPT integration guide

Point to the new guide in test/wpt/README.md instead.

This patch uses the `git node wpt` command in node-core-utils to initialize `test/fixtures/wpt` with a subset of the Web Platform Tests (https://github.com/web-platform-tests/wpt), including: - console tests - url tests - interfaces (.idl WebIDL files) - test harness located in `resources`, including the WebIDL parser Also initializes the README.md, LICENSE and versions.json in the `test/fixtures/wpt`. In later patches, we will use these files to run WPT and move away from the manually maintained copy-pasted files in `test/parallel/test-whatwg-*`. https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt

In favor of actual .any.js tests that will be run with the WPT harness in a subsequent patch. The other types of .js tests (e.g. .window.js tests) and the .html tests remained to be supported or migrated in the upstream.

Removes the following files: - test/fixtures/url-tests.js - test/fixtures/url-setter-tests.js - test/fixtures/url-toascii.js in favor of: - test/fixtures/wpt/url/resources/urltestdata.json - test/fixtures/wpt/url/resources/setters_tests.json - test/fixtures/wpt/url/resources/toascii.json Also removes dependency of `fixtures/url-tests.js` in http2 tests and use `fixtures/person-large.jpg` instead since they are just looking for a big enough file to transfer.

This patch: - Creates a new test suite `wpt` that can be used to run a subset of Web Platform Tests - Adds a `WPTRunner` in `test/common/wpt.js` that can run the WPT subset in `test/fixtures/wpt` with a vm and the WPT harness while taking the status file in `test/wpt/status` into account. Here we use a new format of status file (in JSON) to handle specific requirements (like ICU requirements) in the tests and to handle expected failures and TODOs. - Adds documentation on how the runner and the update automation works - Runs the WHATWG URL tests and the console tests with the new test runner. With this patch we eliminates the need of copy-pasting with manual modifications to update a large chunk of our WPT subset previously maintained in `test/parallel`. Now the tests run in `test/wpt` can be automatically updated with `git node wpt` without modifications by the actual WPT harness instead of our home-grown mock. There are still a few URL tests left that need to be migrated in the upstream to be placed in .js instead of .html - we currently still use the legacy harness mock in the test files.

Point to the new guide in test/wpt/README.md instead.

nodejs-github-bot · 2018-11-02T10:08:03Z

@joyeecheung build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1459/pipeline

joyeecheung · 2018-11-02T10:09:36Z

CI: https://ci.nodejs.org/job/node-test-pull-request/18300/

cc @nodejs/testing @nodejs/url

joyeecheung · 2018-11-02T10:10:56Z

Also maybe @nodejs/open-standards would be interested in taking a look

joyeecheung · 2018-11-02T10:16:59Z

Make linters happy: https://ci.nodejs.org/job/node-test-pull-request/18301/

targos · 2018-11-02T10:44:15Z

Tentative run of wpt on CI: https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/2074/

targos · 2018-11-02T10:54:53Z

06:44:59 + python tools/test.py -j 4 -p tap --logfile test.tap --mode=release --flaky-tests=dontcare wpt
06:45:00 TAP version 13
06:45:00 1..2
06:45:00 ok 1 wpt/test-whatwg-console
06:45:00   ---
06:45:00   duration_ms: 0.332
06:45:00   ...
06:45:00 ok 2 wpt/test-whatwg-url
06:45:00   ---
06:45:00   duration_ms: 0.412
06:45:00   ...

🎉

joyeecheung · 2018-11-02T13:13:26Z

Resumed CI: https://ci.nodejs.org/job/node-test-pull-request/18302/

joyeecheung · 2018-11-06T04:12:48Z

The CI is green. Can I have some reviews please? @nodejs/testing @nodejs/url

Makefile

joyeecheung · 2018-11-08T09:10:08Z

I plan to land this after the 7-day limit passes to continue fixing these failures:

// test/wpt/status/console.json
{
  "idlharness.any.js": {
    "fail": ".table, .dir and .timeLog parameter lengths are wrong"
  }
}

// test/wpt/status/url.json
  "toascii.window.js": {
    "requires": ["intl"],
    "skip": "TODO: port from .window.js"
  },
  "historical.any.js": {
    "requires": ["intl"]
  },
  "urlencoded-parser.any.js": {
    "fail": "missing Request and Response"
  },
  "idlharness.any.js": {
    "fail": "getter/setter names are wrong, etc."
  }
}

And continue porting encoding, timer, performance and messaging (worker even?) tests into this repo without manual copy-pasting - I don't expect all of them to pass but at least it would be nice to know where the inconsistencies are.

So please voice your opinion if you spot something that's just incorrect in theory and cannot be fixed later once this lands (I hope there isn't since this is just home-grown test changes that's completely up to us to fix anytime we want...).

joyeecheung · 2018-11-09T12:36:51Z

Landed in 9d434db...22f3ff9, thanks!

This patch uses the `git node wpt` command in node-core-utils to initialize `test/fixtures/wpt` with a subset of the Web Platform Tests (https://github.com/web-platform-tests/wpt), including: - console tests - url tests - interfaces (.idl WebIDL files) - test harness located in `resources`, including the WebIDL parser Also initializes the README.md, LICENSE and versions.json in the `test/fixtures/wpt`. In later patches, we will use these files to run WPT and move away from the manually maintained copy-pasted files in `test/parallel/test-whatwg-*`. https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

In favor of actual .any.js tests that will be run with the WPT harness in a subsequent patch. The other types of .js tests (e.g. .window.js tests) and the .html tests remained to be supported or migrated in the upstream. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

Removes the following files: - test/fixtures/url-tests.js - test/fixtures/url-setter-tests.js - test/fixtures/url-toascii.js in favor of: - test/fixtures/wpt/url/resources/urltestdata.json - test/fixtures/wpt/url/resources/setters_tests.json - test/fixtures/wpt/url/resources/toascii.json Also removes dependency of `fixtures/url-tests.js` in http2 tests and use `fixtures/person-large.jpg` instead since they are just looking for a big enough file to transfer. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

This patch: - Creates a new test suite `wpt` that can be used to run a subset of Web Platform Tests - Adds a `WPTRunner` in `test/common/wpt.js` that can run the WPT subset in `test/fixtures/wpt` with a vm and the WPT harness while taking the status file in `test/wpt/status` into account. Here we use a new format of status file (in JSON) to handle specific requirements (like ICU requirements) in the tests and to handle expected failures and TODOs. - Adds documentation on how the runner and the update automation works - Runs the WHATWG URL tests and the console tests with the new test runner. With this patch we eliminates the need of copy-pasting with manual modifications to update a large chunk of our WPT subset previously maintained in `test/parallel`. Now the tests run in `test/wpt` can be automatically updated with `git node wpt` without modifications by the actual WPT harness instead of our home-grown mock. There are still a few URL tests left that need to be migrated in the upstream to be placed in .js instead of .html - we currently still use the legacy harness mock in the test files. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

Point to the new guide in test/wpt/README.md instead. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

This patch uses the `git node wpt` command in node-core-utils to initialize `test/fixtures/wpt` with a subset of the Web Platform Tests (https://github.com/web-platform-tests/wpt), including: - console tests - url tests - interfaces (.idl WebIDL files) - test harness located in `resources`, including the WebIDL parser Also initializes the README.md, LICENSE and versions.json in the `test/fixtures/wpt`. In later patches, we will use these files to run WPT and move away from the manually maintained copy-pasted files in `test/parallel/test-whatwg-*`. https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

In favor of actual .any.js tests that will be run with the WPT harness in a subsequent patch. The other types of .js tests (e.g. .window.js tests) and the .html tests remained to be supported or migrated in the upstream. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

Removes the following files: - test/fixtures/url-tests.js - test/fixtures/url-setter-tests.js - test/fixtures/url-toascii.js in favor of: - test/fixtures/wpt/url/resources/urltestdata.json - test/fixtures/wpt/url/resources/setters_tests.json - test/fixtures/wpt/url/resources/toascii.json Also removes dependency of `fixtures/url-tests.js` in http2 tests and use `fixtures/person-large.jpg` instead since they are just looking for a big enough file to transfer. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

This patch: - Creates a new test suite `wpt` that can be used to run a subset of Web Platform Tests - Adds a `WPTRunner` in `test/common/wpt.js` that can run the WPT subset in `test/fixtures/wpt` with a vm and the WPT harness while taking the status file in `test/wpt/status` into account. Here we use a new format of status file (in JSON) to handle specific requirements (like ICU requirements) in the tests and to handle expected failures and TODOs. - Adds documentation on how the runner and the update automation works - Runs the WHATWG URL tests and the console tests with the new test runner. With this patch we eliminates the need of copy-pasting with manual modifications to update a large chunk of our WPT subset previously maintained in `test/parallel`. Now the tests run in `test/wpt` can be automatically updated with `git node wpt` without modifications by the actual WPT harness instead of our home-grown mock. There are still a few URL tests left that need to be migrated in the upstream to be placed in .js instead of .html - we currently still use the legacy harness mock in the test files. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

Point to the new guide in test/wpt/README.md instead. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

This patch uses the `git node wpt` command in node-core-utils to initialize `test/fixtures/wpt` with a subset of the Web Platform Tests (https://github.com/web-platform-tests/wpt), including: - console tests - url tests - interfaces (.idl WebIDL files) - test harness located in `resources`, including the WebIDL parser Also initializes the README.md, LICENSE and versions.json in the `test/fixtures/wpt`. In later patches, we will use these files to run WPT and move away from the manually maintained copy-pasted files in `test/parallel/test-whatwg-*`. https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt PR-URL: nodejs#24035 Refs: nodejs#23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

In favor of actual .any.js tests that will be run with the WPT harness in a subsequent patch. The other types of .js tests (e.g. .window.js tests) and the .html tests remained to be supported or migrated in the upstream. PR-URL: nodejs#24035 Refs: nodejs#23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

Removes the following files: - test/fixtures/url-tests.js - test/fixtures/url-setter-tests.js - test/fixtures/url-toascii.js in favor of: - test/fixtures/wpt/url/resources/urltestdata.json - test/fixtures/wpt/url/resources/setters_tests.json - test/fixtures/wpt/url/resources/toascii.json Also removes dependency of `fixtures/url-tests.js` in http2 tests and use `fixtures/person-large.jpg` instead since they are just looking for a big enough file to transfer. PR-URL: nodejs#24035 Refs: nodejs#23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

This patch: - Creates a new test suite `wpt` that can be used to run a subset of Web Platform Tests - Adds a `WPTRunner` in `test/common/wpt.js` that can run the WPT subset in `test/fixtures/wpt` with a vm and the WPT harness while taking the status file in `test/wpt/status` into account. Here we use a new format of status file (in JSON) to handle specific requirements (like ICU requirements) in the tests and to handle expected failures and TODOs. - Adds documentation on how the runner and the update automation works - Runs the WHATWG URL tests and the console tests with the new test runner. With this patch we eliminates the need of copy-pasting with manual modifications to update a large chunk of our WPT subset previously maintained in `test/parallel`. Now the tests run in `test/wpt` can be automatically updated with `git node wpt` without modifications by the actual WPT harness instead of our home-grown mock. There are still a few URL tests left that need to be migrated in the upstream to be placed in .js instead of .html - we currently still use the legacy harness mock in the test files. PR-URL: nodejs#24035 Refs: nodejs#23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

Point to the new guide in test/wpt/README.md instead. PR-URL: nodejs#24035 Refs: nodejs#23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

codebytere · 2018-12-13T22:09:42Z

@joyeecheung this doesn't land cleanly on v10.x; if you think it's a backport candidate could you please backport it manually?

joyeecheung added 5 commits November 2, 2018 17:45

test: remove WPT tests that are now .any.js in the upstream

531d0e2

In favor of actual .any.js tests that will be run with the WPT harness in a subsequent patch. The other types of .js tests (e.g. .window.js tests) and the .html tests remained to be supported or migrated in the upstream.

doc: remove legacy WPT integration guide

c6a7738

Point to the new guide in test/wpt/README.md instead.

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Nov 2, 2018

joyeecheung added 2 commits November 2, 2018 18:15

fixup! test: initialize test/wpt to run URL and console .js tests

d82e25b

fixup! doc: remove legacy WPT integration guide

b9e0036

joyeecheung mentioned this pull request Nov 6, 2018

Let's do this nodejs/whatwg-stream#1

Open

watilde reviewed Nov 7, 2018

View reviewed changes

Makefile Show resolved Hide resolved

joyeecheung mentioned this pull request Nov 7, 2018

Consider making URL's context symbol more private, or at least resetting the "flags" member #24211

Closed

watilde approved these changes Nov 8, 2018

View reviewed changes

joyeecheung closed this Nov 9, 2018

joyeecheung added a commit that referenced this pull request Nov 9, 2018

doc: remove legacy WPT integration guide

22f3ff9

Point to the new guide in test/wpt/README.md instead. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

joyeecheung mentioned this pull request Nov 9, 2018

[Tracking Issue] Web Platform Tests revamp/update automation #23192

Closed

14 tasks

BridgeAR pushed a commit that referenced this pull request Nov 14, 2018

doc: remove legacy WPT integration guide

7bcc4cc

Point to the new guide in test/wpt/README.md instead. PR-URL: #24035 Refs: #23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

BridgeAR mentioned this pull request Nov 14, 2018

Release proposal: v11.2 #24350

Merged

kiyomizumia pushed a commit to kiyomizumia/node that referenced this pull request Nov 15, 2018

doc: remove legacy WPT integration guide

df2ab63

Point to the new guide in test/wpt/README.md instead. PR-URL: nodejs#24035 Refs: nodejs#23192 Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>

codebytere added the backport-requested-v10.x label Dec 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: automate WPT updates with drivers in test/wpt #24035

test: automate WPT updates with drivers in test/wpt #24035

joyeecheung commented Nov 2, 2018 •

edited

Loading

nodejs-github-bot commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

targos commented Nov 2, 2018

targos commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

joyeecheung commented Nov 6, 2018

joyeecheung commented Nov 8, 2018 •

edited

Loading

joyeecheung commented Nov 9, 2018

codebytere commented Dec 13, 2018

test: automate WPT updates with drivers in test/wpt #24035

test: automate WPT updates with drivers in test/wpt #24035

Conversation

joyeecheung commented Nov 2, 2018 • edited Loading

test: use git node wpt to pull WPT into test/fixtures

test: remove WPT tests that are now .any.js in the upstream

test: use URL fixtures under test/fixtures/wpt/url/resources

test: initialize test/wpt to run URL and console .js tests

doc: remove legacy WPT integration guide

nodejs-github-bot commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

targos commented Nov 2, 2018

targos commented Nov 2, 2018

joyeecheung commented Nov 2, 2018

joyeecheung commented Nov 6, 2018

joyeecheung commented Nov 8, 2018 • edited Loading

joyeecheung commented Nov 9, 2018

codebytere commented Dec 13, 2018

joyeecheung commented Nov 2, 2018 •

edited

Loading

joyeecheung commented Nov 8, 2018 •

edited

Loading